parallel article building via n_cores#2989
Open
billdenney wants to merge 7 commits intor-lib:mainfrom
Open
Conversation
Add `n_cores` argument to `build_articles()`, `build_site()`, and `build_site_github_pages()` so articles can be rendered in parallel via `purrr::in_parallel()` + mirai. Default `n_cores = 1L` preserves the existing serial code path and requires no new packages; `Inf` autodetects via `parallel::detectCores()`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
`parallel::detectCores()` is used by `check_n_cores()` when `n_cores = Inf`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Replace `pkgdown:::quarto_article_postprocess(...)` with
`utils::getFromNamespace("quarto_article_postprocess", "pkgdown")`
inside the `purrr::in_parallel()` worker closure. This clears the
R CMD check NOTE about calls to own-namespace `:::`.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* build-quarto-articles: pandoc added a default-styles comment header to its CSS output. * build-reference: evaluate/knitr/rmarkdown now consumes two extra RNG draws before the example runs, shifting the output by two values. Both are environmental drift, not behavioral changes from this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
`purrr::in_parallel()` uses `carrier::crate()` to serialize worker closures and raises an undeclared-dependency error partway through if carrier isn't installed. Add carrier to Suggests and to the `rlang::check_installed()` pre-flight so the requirement is surfaced up-front with pkgdown's own error path. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
`in_parallel(.f, ...)` passes the iteration element(s) to `.f` and captures the extra `...` bindings into `.f`'s environment via `carrier::crate()`, not into its formals. The previous closures listed `pkg`, `lazy`, `seed`, `quiet` as arguments, so when purrr invoked them with only the iteration variable the additional formals were unbound and raised "argument pkg is missing". Drop the extras from the formals and reference them as free variables in the body; the carrier crate supplies them. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
When `install = TRUE`, pkgdown installs the package into a `withr::local_temp_libpaths()` directory visible only to the main R session. mirai workers are fresh subprocesses that default to the system `.libPaths()` and cannot see the temp install, so any article that depends on dev-only exports of the host package fails on the worker. Capture `.libPaths()` at dispatch time and re-set it inside each worker call via the `carrier` crate. This costs one cheap call per article and keeps the parallel path honest for `install = TRUE`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Author
|
One additional consideration is: Do we also want to parallelize the reference building? That generally seems to be fast, so it didn't seem to be differentiated to me. |
Author
|
I also updated the snaps here to ensure that everything was building cleanly, thought part of the snap update is not specific to this feature. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
n_coresargument tobuild_articles(),build_site(), andbuild_site_github_pages()for parallel vignette rendering viapurrr::in_parallel()+ mirai.n_cores = 1Lpreserves current serial behavior (no new hard deps).Infautodetects workers viaparallel::detectCores(); fractional values round up.Notes for reviewers
purrrto>= 1.1.0(forin_parallel()) and addsmiraito Suggests — only required whenn_cores > 1.walk2is also parallelized for symmetry; happy to narrow to the Rmd path if preferred — that's where the real wall-time win is.Test plan
devtools::document()cleandevtools::test(filter = "build-(articles|site|github)")passesbuild_articles(n_cores = 2)on a package with ≥2 Rmd vignettesFix #2988